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Abstract 

As part of a continuing effort to re-engineer the 
wind-tunnel testing process, a comprehensive data 
quality assurance program is being established at NASA 
Langley Research Center (LaRC). The ultimate goal of 
the program is routine provision of tunnel-to-tunnel 
reproducibility with total uncertainty levels acceptable 
for test and evaluation of civilian transports. The 
operational elements for reaching such levels of 
reproducibility are: (1) statistical control , which 

provides long-term measurement uncertainty 
predictability and a base for continuous improvement, 
(2) measurement uncertainty prediction , which provides 
test designs which can meet data quality expectations 
within the system’s predictable variation, and (3) 
national standards , which provide a means for resolving 
tunnel-to-tunnel differences. The paper presents the 
LaRC design for the program and discusses the process 
of implementation. 

Introduction 

About three years ago, the NASA Langley 
Research Center (LaRC) wind-tunnel establishment 
committed itself to a cultural shift from largely 
supporting the internal research of individual Branches, 
each organized around one or more wind tunnels, to 
supporting national programs developed in partnership 
by various NASA centers, other government 
organizations and industry. This shift has required 
entirely new levels of customer trust in the results 
produced by the tunnels 1 and led to consolidation of 
tunnel assets, originally involving eight different 
facility cultures in five Branches (Figure 1), into a 
single Research Facilities Branch (RFB) in the 
Aerodynamics Division (AD). 

The consolidation was accompanied by a far- 
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reaching change agent, Re-Engineering Wind Tunnel 
Testing at LaRC. The re-engineering effort 2,3 has been 
specifically tasked with developing a uniform customer- 
focused culture, increasing tunnel productivity, reducing 
costs and test process time, and improving data quality* 
to levels suitable for national standards. 

Out of the initial re-engineering design work, five 
major efforts having an influence on data quality 
assurance DQA) were established: 

• Test Processes 

• Information Management 

• Wall Interference and Correction 

• Facility Operations and Implementation 

• Measurement Uncertainty 

Each team is accountable for identifying current practice 
for, standardizing, documenting and improving all of the 
processes assigned to it. The teams are also responsible 
for any training and certification required for the 
processes for which they are accountable. In this paper, 
I will describe RFB work involved in measurement 
uncertainty and statistical control only, although all of 
the re-engineering effort, including the work of several 
other divisions which support the wind tunnels, has an 
impact on the data quality assurance (DQA) program. 

The key elements of the LaRC DQA program, as it 
is understood now, are presented in the next section. 
The program consists of four phases (Figure 2) which 
have been considerably revised and expanded during the 
two years since the beginning of the present level of 
effort as the participants have learned more about 
measurement uncertainty practice in the real world of 
high-productivity wind-tunnel testing and have begun to 
determine the process changes which appear to be 
required to achieve the program goals. 

The Key Elements description is followed by four 


t For the purposes of this paper, the measurement 
uncertainty goals are the levels which would be acceptable 
for test and evaluation of civilian transports. For example, 
typical single-test total uncertainty requirements for the 
lift, pitching-moment and drag coefficients at transonic 
cruise conditions might be ±0.01, ±0.001 and ±0.0001 
respectively. 
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sections describing the implementation so far of each 
phase of the project. The paper concludes with a final 
comment. The nomenclature used is intended to be 
consistent with the definitions used in references 4-7. 

Key Elements of the Program 

Statistic al Control 

The operational philosophy underlying the new 
program is the measurement process as taught by 
Churchill Eisenhart 8 and his colleagues 69 at the former 
National Bureau of Standards (NBS), now the National 
Institute of Standards and Technology (NIST). 
Eisenhart’s distinction of a measurement process is 
derived from the statistical control and continuous 
improvement work of Shewhart 10 and his famous 
successor, W. Edwards Deming 11 . Although Eisenhart 
was referring to instrument calibration systems 8 , his 
definition applies just as well to the system composed 
of the wind tunnel, its instrumentation and data 
systems, the personnel running the systems and 
performing the procedures, and the metrology support 
system. Since he defined the distinction so eloquently 8 , 
I will simply quote him here: 

“Measurement is ordinarily a repeatable 
operation, so that it is appropriate to regard 
measurement as a production process, the 
‘product’ being the numbers, i.e., the 
measurements, that it yields; and to apply to 
measurement processes in the laboratory the 
concepts and techniques of statistical process 
control that have proved so useful in the quality 
control of industrial production. 

“Viewed thus it becomes evident that a 
particular measurement operation cannot be 
regarded as constituting a measurement process 
unless statistical stability of the type known as 
a state of statistical control has been attained” 

(The italics are mine.) In reference 6, Taylor and 
Oppermann of NIST define statistical control to be “ the 
attainment of a state of predictability [such that] the 
mean of a large number of measurements will approach 
a limiting value (limiting mean) and the individual 
measurements should have a stable distribution, 
described by their standard deviation. ” 

Clearly, wind-tunnel testing is a process and all 
processes have inherent variation. Data quality 
assurance is basically the structure with which that 
process is managed to produce measurement uncertainty 
levels which are in statistical control within the limits 
desired. The biggest advantage of this approach is that 
such a measurement process is predictable. 


For the LaRC DQA program, the core 
measurement process is the acquisition of a data polar, 
i.e. a sequence of data points taken in a prescribed 
manner with just one of the independent variables 
changing. The most typical polar in RFB is an angle- 
of-attack sweep with angle of sideslip, Mach number, 
Reynolds number and configurational geometry held 
nominally constant. Following the definition given in 
reference 7, the repeatability of this process is found by 
repeating the polar over a short time with no changes to 
the system. Hence, the repeatability is found by 
obtaining back-to-back polars with unchanged 
conditions except for the cycling of the one independent 
variable which is allowed to change. 

Such repeatability, of course, is insufficient for 
either incremental or absolute testing and a hierarchical 
sequence of such processes yielding the desired 
reproducibility 7 over time and space must be considered. 
The reproducibility sequence that LaRC intends to use 
to establish and maintain statistical control is given in 
Figure 3. Back-to-back polars and repeats of those data 
sets over the course of a test entry are not sufficient to 
establish statistical control for the rest of the 
reproducibility hierarchy since the limiting mean would 
change with each model tested. Hence, it is necessary 
to use a check standard model in each tunnel to 
determine reproducibility for multiple entries and 
national check standards for establishing tunnel-to- 
tunnel reproducibility. This approach is, of course, just 
standard practice at a NIST-qualified laboratory. 6 

A secondary, but no less valuable, advantage of the 
statistical control approach is that economical and 
predictable continuous improvement can be planned and 
carried out. 

Measurement Uncertainty Prediction (and Reporting) 

The second critical element of the LaRC program is 
an insistance on measurement uncertainty prediction 
early enough in the test planning process to allow for a 
proper selection of instrumentation, test strategy, and, if 
necessary, test objectives. 4,12,13 The importance of 
credible prediction in the satisfaction of customer 
expectations cannot be emphasized too strongly. 

National Standards 

The third critical element of the LaRC program 
arises from the realization that the ultimate goal for a 
wind-tunnel measurement uncertainty standard must be 
tunnel-to-tunnel reproducibility for the same model 
(national check standard) corrected to the same free-air 
conditions. Resolution of any discrepancies will 
involve not only credible statements of measurement 
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uncertainty for each tunnel engaged in the comparison 
but also will demand some sort of agreement (i.e. 
national standards) to resolve residuals. 

Implementation Phases 

With the above three legs driving the program, 
together with a need to get interim results as soon as 
possible, not only for customer negotiations but also to 
help support the creation of a uniform culture oriented 
toward statistical thinking and customer satisfaction, the 
decision was made to divide the program 

implementation into the following four phases 

(Figure 2): 

I. Simplified Uncertainty Prediction and 

Reporting 

II. Detailed Uncertainty Analysis and Statistical 

Control 

III. Certification and National Standards 

IV. Continuous Improvement 

with work presently proceeding on Phases I and II 
simultaneously. As part of Phase II, the Test Processes 
Team is working to delineate all of the important 
processes involved in the wind-tunnel testing enterprise 
with the goal being to determine what’s so now and to 
eventually standardize the processes for the purpose of 
statistical control as meant by Eisenhart, Shewhart and 
Deming. Of course, RFB’s experience is that the sum 
total of processes for each tunnel has attained some 
level of statistical control even if it is not known 
exactly what that level is. So the program participants 
are refraining from “tampering” (in the sense of 
Deming 14 ) with the present processes affecting 
measurement uncertainty until the work of the Test 
Processes Team is complete. 

Phase I - Simplified Uncertainty 
Prediction and Reporting 

Phase I could also be called “Getting Started.” 
Although considerable effort 15 * 20 in the areas of 
assessing measurement uncertainty and improving data 
quality had been conducted in the National Transonic 
Facility prior to the formation of the Research Facilities 
Branch and the start of the present program, no 
comprehensive systemic changes had been implemented. 
Adopting the point of view that statistical thinking and 
designing wind-tunnel tests using uncertainty analysis 
are fundamental paradigm shifts, 21 the program 
participants elected to proceed using a bootstrap process 
somewhat similar to historical cultural shifts in the area 
of precision measurement. 22 Such shifts have usually 
proceeded through phases similar to those adopted for 


the LaRC program: (1) recognition of the need and the 
first, somewhat crude (in hindsight), efforts, 

(2) increased attention on the complexity of the problem 
and improved analysis and control of the process, 

(3) resolution of lingering conflicts through the 
development of standards, and (4) continual 
improvement as understanding increases and better 
instrumentation becomes available. 

It should be understood that the personnel 
developing and implementing the program had either no 
prior or only modest experience with or training in 
statistical thinking 8 11,14 or modem measurement 
uncertainty. 4,12,13,23 Hence, the bootstrap process also 
allowed personnel to implement first-order assessment 
methods while simultaneously learning appropriate 
strategies and gathering increasingly detailed 
information about the tunnel systems and procedures. 

The actual steps followed in Phase I were as 
follows: 

1. Presentations of the short course, 
“Experimentation and Uncertainty 
Analysis,” by Hugh W. Coleman and W. 
Glenn Steele, Jr., to all RFB test engineers 
and data specialists and about half of the AD 
researchers. 

2. Application of General Uncertainty 
Analysis 4 to specific tests with difficult 
measurement uncertainty requirements. 

3. Creation of spreadsheets for each tunnel for 
prediction of total uncertainty and single-test 
repeatability based on a simplified analysis. 

4. Adoption of an interim RFB Data Quality 
Prediction and Assessment Procedure. 

Presentation of Short Course 

The short course presentations in the Summer and 
Fall of 1994 brought about a general increase in 
awareness of the subject of measurement uncertainty and 
the distinctions of bias, precision and uncertainty 
propagation for derived quantities. However, the 
heightened awareness did not bring about a general 
increase in measurement uncertainty activity and it 
became clear that training without application and 
appropriate management direction would be insufficient. 

A pplication of General Uncertainty Analysis to Specific 
Tests 

In the Fall of 1994 and the Spring of 1995, four 
tests in three different tunnels were supported with 
simplified uncertainty analysis predictions in the spirit 
of the General Uncertainty Analysis for test planning 
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suggested in Chapter 3 of reference 4. Such an analysis 
ignores the details of the bias and precision components 
and concentrates on the effect of each instrument’s 
overall contribution to the uncertainty of the derived 
coefficients of interest. Despite the simplified nature of 
the prediction methodology, it was clear to the test 
engineers and customers that the chosen instrumentation 
for each test would be inadequate to achieve the stated 
test objectives. In two cases, the instrumentation was 
changed. In the rest, the test objectives were changed. 

The importance of these initial applications of the 
simplified analysis should not be underestimated. The 
results led to a heightened awareness of the value of 
predictions however crude, to a sense of urgency in 
program implementation, and, ultimately, to the 
decision to implement as quickly as possible the 
simplified methodolgy for prediction and reporting for 
all of the RFB tunnels. It is understood that this 
interim approach does not conform to present standards 
for reporting measurement uncertainty, 7, 23 25 but it does 
allow LaRC to derive considerable benefits from its 
present, immature, knowledge while the detailed 
information and statistical control needed for reporting 
according to the standards is obtained. 

Spreadsheets for Simplified Analysis 

The uncertainty prediction spreadsheets are based on 
the notion of a General Uncertainty Analysis 4 as noted 
in the previous subsection. The spreadsheets are based 
on a simple model of the data reduction process for 
measurement of forces and moments corrected for base 
and cavity pressure effects, but not corrected for wall or 
support interference. For example, the instruments 
included in the simple model for a typical transonic 
tunnel test are the six components of the force balance, 
the angle-of-attack sensor, the base and cavity pressure 
sensors, the tunnel total pressure, the static pressure in 
the plenum which surrounds the slotted test section, and 
the tunnel total temperature. For total uncertainty 
predictions, bias and precision limits for each 
instrument are lumped together in a single number 
estimated by agreement of the test engineers in each 
facility. The spreadsheets display on each page the 
effect of the uncertainty of each instrument, reference 
quantity and transfer distance on the uncertainties of the 
tunnel parameters and the derived coefficients of interest 
for a single set-point condition. 

The effect of the total uncertainty prediction is 
mostly tutorial, i.e. it serves to educate the customer 
and test engineer to probable test-to-test and tunnel -to- 
tunnel reproducibility problems. The same spreadsheets 
are also used to estimate single-test reproducibility. 
These predictions are presently made for three reasons: 


(1) some of RFB* s customers have specific single-test 
reproducibility requirements and need to be informed if 
the selected instrumentation and test conditions will 
meet their objectives, (2) a set of reproducibility 
measurements made during the test can be used to test 
and update the simplified prediction model, and (3) the 
knowledge that a test’s uncertainty objectives are 
difficult but possible to meet based on previous facility 
experience alerts test personnel to use procedures which 
are known to produce the best data quality but which 
may consume significantly more time. It has not been 
uncommon for a customer to have to accept a reduction 
in the size of the test matrix and/or a change in the test 
conditions to meet uncertainty objectives. 

Interim Data Quality Assessment Procedure 

It is the interim policy of the Research Facilities 
Branch that the following four types of simplified 
analyses should be performed for each wind-tunnel test: 

1 . Pre-Test Prediction of Total Uncertainty - Use 
the simplified uncertainty propagation equations (see 
previous subsection), together with estimates of the 
total uncertainty for each primary measurement process 
in the equations. This analysis is especially important 
for customers who are interested in using the data for 
CFD validation, for development of data bases for 
simulations, or for test technique development, e.g. 
semispan testing. 

2. Pre-Test Prediction of Reproducibility During a 
Single Test Entry - Use the simplified uncertainty 
propagation equations, together with estimates of the 
precision for each primary measurement process in the 
equations. This analysis estimates the reproducibility 
likely to be achieved during the test, especially if it is 
based on historical data for that tunnel. 

3. Post-Test Reproducibility Analysis - Analyze 
the set of repeat polars obtained during the test so that 
the range of variation can be estimated to one 
significant figure. This result, and how it was obtained, 
should be included in the data transmittal and stored in 
the tunnel archives for interim tracking of statistical 
control. The suggested repeat-polar schedule is to 
obtain three back-to-back polars at the beginning, 
middle and end of the test ( nine polars in all) on a 
baseline configuration and a set of test conditions of 
greatest interest to the customer. If a suitable repeat- 
polar set is not obtained, no uncertainty statement will 
be included in the data transmission to the customer. 

4. Post-Test Statement of Bias Limits - Use the 
simplified uncertainty propagation equations to estimate 
the bias limit for each derived quantity of interest 
(usually the balance coefficients, Mach number, 
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Reynolds number and angle of attack) using the best 
available guess of the measurement process biases. 
This information should be included on the data 
transmittal file if the customer so desires. (See caveat in 
previous paragraph.) 

Phase IT - Detailed Uncertainty Analysis 
and Statistical Control 

The purpose of a detailed uncertainty analysis is to 
investigate the contributions of the individual bias and 
precision error sources and combine them into overall 
bias and precision limits for the (derived) experimental 
results of interest. 4 This is a huge task for a typical 
wind-tunnel operation and the LaRC program has just 
begun to go beyond the simplified analyses described 
above. Batill 15 conservatively estimated that 250,000 
pieces of information are acquired and processed to make 
a single drag-coefficient calculation at the National 
Transonic Facility. Consequently, it seems appropriate 
to continue the bootstrapping process originally 
adopted. In other words, the program will attempt to 
obtain an increasingly more accurate model of the 
measurement process in an iterative manner, considering 
the most obvious sources of variability first. 26 Of 
course, such models will be meaningless if statistical 
control is not demonstrated for each level considered. 9 

The four major elements of the RFB plan 
(Figure 4) for achieving and monitoring statistical 
control are (1) tunnel calibration, (2) test-section flow 
characterization, (3) characterization of wall and support 
interference, and (4) characterization of precision and 
bias limits. The individual elements are discussed in the 
following subsections, together with appropriate time 
scales for each activity (Figure 4). 


For statistical control of the tunnel, measurements 
should be taken at least annually to check the 
calibrations. 27 In this regard, LaRC intends to use 
centerline pitot-static probes, following the approach 
used at the Boeing Transonic Wind Tunnel. 27 The 
probes should be considered to be working standards and 
should be treated with great care. 27 

Test-Section Flow Characterization 

Figure 6 shows four flow-characterization activites 
that are considered to be important in the LaRC DQA 
program for statistical control and measurement 
uncertainty. The use of model check standards to 
annually obtain upwash and sidewash on the tunnel 
centerline and the location of boundary-layer transition 
help to establish test-to-test reproducibility levels and 
verify statistical control. Once an historical data base 
has been obtained, the annual measurements can be used 
to determine if action is required to clean the screens, 
etc., during the next maintenance period. 27 

Test-section flow-field surveys, conducted on an 
infrequent basis (see Figure 4), are helpful for a variety 
of experiments which are not conducted on the tunnel 
centerline. For example, the results can be used to 
make rough estimates of bias due to local flow 
angularity and can point to possible improvements in 
the flow-straightening system. 

Although no such effort has been yet established at 
LaRC, it is believed that on-line monitoring of flow 
quality at selected locations would be helpful in 
determining critical aspects of process variation and 
point to possible out-of-statistical-control situations. 

Characterization of Wall and Support Interference 


There are many activities associated with what is 
called “Tunnel Calibration” at LaRC. But here we shall 
consider only those measurements which are connected 
with determining and monitoring the calibration 
constants for derivation of the test-section Mach number 
and dynamic pressure and for estimating the empty test- 
section axial pressure gradients (Figure 5). For 
subsonic and transonic tunnels, the calibration constants 
and axial pressure gradients would be inferred from 
infrequent (every five to ten years) centerline pipe 
measurements. In the past, such measurements would 
be obtained once for each set of tunnel conditions of 
interest. In the future, such measurements should be 
repeated over several tunnel and data acquisition system 
cycles to help determine the uncertainty of the 
constants. 


A full discussion of present LaRC activities in wall 
and support interference is beyond the scope of this 
paper. However, it is clear that certain statistical 
control activities are required (Figure 7). First, it is 
imperative that the correction models used be checked 
on an infrequent basis (Figure 4) by testing different 
sizes of some check standard. Secondly, the 
reproducibility of the corrections should be checked 
frequently. This entails testing both with the tunnel 
empty and with a check standard installed. 

Characterization of Precision and Bias 

Reproducibility sufficient for a credible 
measurement uncertainty statement depends, at a 
minimum, on traceable configuration control and repeat- 
polar measurements during a test entry as shown in 
Figure 8. It seems also rather clear that frequent tests of 
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a check standard and regular recalls of instrumentation 
are necessary for statistical control. But it should also 
be apparent that some kind of statistical control must be 
established for the execution of procedures by test 
personnel. In this regard, standardization of procedures, 
clear and easy-to-use documentation, and frequent 
training and certification are necessary. If possible, 
regular random checks on the output of critical 
procedures should be conducted. If a range of 
individuals are involved in the execution of a given 
procedure, it is important that the variation across the 
set of individuals be established. 

Phase III - Ce rtification and Na tional Standards 

Although no significant work has been conducted 
in Phase III, it is believed that the elements shown in 
Figure 9 are required, as a minimum, to establish 
credible tunnel-to-tunnel reproducibility. To the 
author’s knowledge, nothing like this in the area of 
wind-tunnel testing has been achieved before. Yet it 
seems essential if today’s data quality requirements are 
to be met. Any effort in this area will likely require 
that credible statistical control, according to some kind 
of national standard, be established for each tunnel 
participating. Furthermore, a committee would have to 
be created to resolve the residuals left after the testing of 
national check standards. The resolution would 
probably lead to the selection of a composite standard to 
which all of the tunnels are biased by a known 
correction together with its uncertainty. 

Phase IV - Continuous Improvement 

At the end of Phases I and II and statistical control 
will have been achieved, the data quality limits of 
present instrumentation and processes will have been 
reached and further improvement in data quality levels is 
unlikely without fundamental changes in processes or 
major instrumentation. In this regard, the author 
suggests using the Shewhart Cycle 14,28 for rational, 
economical, continuous improvement (see Figure 10). 
The value of the Shewhart approach (known as the 
Deming Cycle in Japan) has been demonstrated 
repeatedly in complex industrial processes and should 
prove just as valuable for wind-tunnel testing. 

Final Comment 

The Data Quality Assurance program described in 
this paper is composed of many elements, all of which 
have been described previously in the literature or are 
being used in other facilities (e.g. the Boeing 
Company 27 ). Hence, none of them are new. The chief 


contribution of the LaRC program has been to use 
simplified analyses to get started and to emphasize 
statistical control in the manner promoted by the 
National Institute of Standards and Technology. 

Having said that, I feel compelled to underscore 
here several points made in the paper: 

1 . The implementation (and even conception) of a 
data quality program of the type described here involves 
a cultural change of huge proportions. The problem is 
that, at the beginning, it will not appear to be a shift 
(and, hence, require new thinking) to the participants. 
Consequently, training by itself is insufficient. All 
involved will have to apply the ideas and methods in 
various situations to apprehend the purposes and 
techniques of the program. 

2. Although the major authors in the engineering 
application of measurement uncertainty 412,13,23 have 
repeatedly stressed the value of uncertainty predictions 
in the design of experiments, it is not yet an idea whose 
time has come. Let me just say that it has been the 
program participants’ experience that even the simplest 
measurement uncertainty analysis produces value far 
beyond the effort required. 

3. Viewed under the Eisenhart, et al, spotlight, it 
becomes obvious that changes in wind-tunnel testing 
measurement uncertainty levels for a system in 
statistical control cannot be produced on demand (e.g. 
“Quality by Exhortation”). Since credible measurement 
uncertainty levels are a product of a process in statistical 
control, they can only be improved by making systemic 
changes in the process itself. 10,11 
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Figure 1. - 1994 Consolidation of LaRC Aerodynamics Division Wind Tunnels. 
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Figure 2. - Implementation Phases of LaRC DQA Program. 

















Figure 3. - Hierarchy of Reproducibility Levels Required 
for Incremental and Absolute Testing 
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Figure 4. - Phase II - Major Statistical Control Activities and 
Definition of Associated Time Scales. 










Figure 5. - Elements of Tunnel Calibration Activity 



Figure 6. - Elements of Test-Section Flow Characterization Activity 
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Figure 7. - Elements of Characterization and Tracking of 
Wall and Support Interference 
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Figure 8. - Elements of Continuous Characterization and 
Tracking of Precision and Bias 














Figure 9. - Phase III - Creation of National Standards and 
Tunnel-to-Tunnel Reproducibility 


Organization's 



Figure 10. - Phase IV - Implement Shewhart/Deming Cycle for 
Continuous Improvement. 












